Evaluation of Speech Synthesis Systems using the Speech Reception Threshold Methodology

The intelligibility of speech sysnthesis systems that are available nowadays is usually high enough to enable comparisons between different synthesis systems based on the speech quality. However, in some situations, like a civil aircraft cockpit, the acoustic environment may be such that intelligibility is a discriminating factor between systems. In this paper we propose a methodology for comparing speech synthesis systems based on the Speech Reception Threshold (SRT). With this method the signal-to-noise ratio is found at which 50% intelligibility of redundant sentences is reached. A system with a lower SRT value is said to be more robust against masking noise. We have compared 5 commercial speech synthesis systems (4 male voices, 5 female voices) in an SRT experiment using a masking noise that was spectrally equivalent to cockpit noise. SRT values range from -4.1dB to 1.1dB. An ANOVA revealed that two of the nine systems had a significantly lower SRT than the rest. There was also an effect of the test subject, which is remarkable because the SRT has usually small variability over listeners. INTRODUCTION The EU 6th framework programme project SAFESOUND involves research to implement modern speech and audio technologies in the cockpit of a civil airline aircraft, in order to enhance the safety of flights. One of the subsystems implements Direct Voice Output (DVO) in order to communicate system messages to the pilots. For the DVO system a commercial-off-the-shelf (COTS) speech synthesis system is envisaged. The main selection criterium for the system is maximum intelligibility of the messages in the noisy civil aircraft cockpit. The quality of commercially available speech synthesis systems is considered to be high enough that for most applications the intelligibility of the systems is not an issue, but rather other performance measures such as speech quality or listening comfort. However, for applications that should increase safety in a noisy cockpit environment, we argue that intelligibility is of highest concern in the choice of a synthesis system, since the mis-interpretation of a spoken system message can have negative consequences to the safety of the aircraft. Several methodologies for assessing the intelligibility can be found in literature, e.g., diagnostic rhyme test (DRT, see Voiers, 1977), or the Diphone Test (Pols et al., 1987; van Bezooijen and Pols, 1987). For a good overview of these methodologies, see van Bezooijen and van Heuven (1997). These methodologies were developed at a time when the technology of speech synthesis systems was not as advanced as it is nowadays. Then, typical systems were formant-based or diphone-based, while modern systems are based on unit selection. In these methodologies ambient noise is not the key factor for determining the intelligibility, but rather the specific technology that is used. We therefore investigated new methodologies and came up with extending the use of the Speech Reception Threshold (SRT) to speech synthesis systems. The SRT methodology uses short redundant sentences as speech stimulus material that subjective listeners are exposed to. The sentences are linguistically meaningful, and consist of 8-9 syllables. This type of van Leeuwen, D.A.; van Balken, J. (2005) Evaluation of Speech Synthesis Systems using the Speech Reception Threshold Methodology. In New Directions for Improving Audio Effectiveness (pp. 9-1 – 9-6). Meeting Proceedings RTO-MP-HFM-123, Paper 9. Neuilly-sur-Seine, France: RTO. Available from: http://www.rto.nato.int/abstracts.aps. Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 01 APR 2005 2. REPORT TYPE N/A 3. DATES COVERED 4. TITLE AND SUBTITLE Evaluation of Speech Synthesis Systems using the Speech Reception Threshold Methodology 5a. CONTRACT NUMBER